Estimating Conditional Densities from Sparse Data for Statistical Language Modeling

The Maximum Likelihood Set (MLS) was recently introduced in [1] as an effective, parameter-free technique for estimating a probability mass function (pmf) from sparse data. The MLS contains all pmfs that assign merely a higher likelihood to the observed counts than to any other set of counts, for the same sample size. In this paper, the MLS is extended to the case of conditional density estimation. First, it is shown that, when the criterion for selecting a pmf from the MLS is the KL-divergence, the selected conditional pmf naturally has a back-off form, except for a ceiling on the probability of high frequency unigrams that are not seen in particular contexts. Second, the pmf has a sparse parameterization, leading to efficient algorithms for KLdivergence minimization. Finally, a novel fattening of the MLS, called the High Likelihood Set (HLS) is introduced. It contains the MLS, and some neighboring pmfs. Experimental results from bigram and trigram estimation indicate that pmfs selected from the HLS are competitive with state-of-the-art estimates. I. THE DENSITY ESTIMATION PROBLEM The problem of probability density estimation may be formulated as follows: a sequence W = {w1, . . . , wN} of independent samples, drawn according to an unknown probability mass function (pmf) PTrue is observed, and the goal is to estimate PTrue. It is assumed that the samples wj belong to a discrete and finite set V = {1, . . . , V }. To facilitate a more concrete exposition, think of V as the vocabulary of a statistical language model (LM), and W as the training corpus. This estimation problem is, of course, a recurring problem not only in natural language processing (NLP) but indeed in all of statistics. A popular estimate of PTrue is the maximum likelihood estimate,